1. Field of the Invention
The present invention is directed to a method and apparatus which provides very simple segmentation, control and interconnection of any neural network and, more particularly, to a system which segments a digital neural network according to its interconnectivity and provides a daisy chained control signal for each segment together with time-division multiplexed communication buses connected to the nodes in the segment.
2. Description of the Related Art
Neural net computing structures are very useful for modeling non-linear problems. Such structures have found wide application in many fields. However, the use of neural nets in real applications is constrained because of their implementation difficulty. General purpose and special purpose computing machines which can simulate the action of a neural net are readily available but are satisfactory only for low performance systems. Current computing technology cannot meet the needs for high speed neural net implementations.
Attempts to build neural net structures directly (rather than simulate such nets with a general purpose computer) face two fundamental problems: size and interconnectivity. Neural nets tend to be very large and very highly interconnected. Therefore, a successful neural net implementation must find a way to build each node of a neural net very efficiently and find a way to connect them very efficiently.
Each node in a neural net implements a transfer function which has many inputs and a single output. The transfer function can be written as: EQU F.sub.N =f(x.sub.0,x.sub.1,x.sub.2,x.sub.3 . . . x.sub.n) (1)
where the subscripted x parameters represent the input values to the node. Each node in a neural net does not have to implement the same function However, one common approach to the design of neural nets does implement the same function in every node. This function has the form: ##EQU1## Other neural network nodes perform a function such as: ##EQU2## and still others compare the right side of equation (3) to a threshold to determine whether to produce on output.
Neural networks 2 are often represented as shown in FIG. 1. Each circle represents one node and the arrows show the flow of inputs and outputs from one node to another. The input nodes do not implement any mathematical operation and simply serve to show the flow of data.
The most common interconnection scheme for neural networks 4 is shown in FIG. 2. In this approach a net comprises three layers, an input layer 6, a second layer 8 (usually called the hidden layer) which receives data from the input layer, and an output layer 10 which receives data from the hidden layer 8 and generates output values. In its classic form, this approach requires that every node in the hidden layer 8 receive data from every input node, and that every output node receive data from every hidden node. Each node implements the identical function. For this classic approach, it is easy to see that the number of interconnects will be the sum of the products of each successive pair of layers. If every layer has one thousand nodes, there will be two million interconnects.
It is important to note that not every neural network is of the form in FIG. 2. Some networks are much more generic and permit any set of interconnections to be made between any set of nodes, and for any node to implement any function. FIG. 3 shows a network 14 of identical nodes which are totally interconnected (a Hopfield net) and FIG. 4 shows an arbitrarily connected network 16 with seven arbitrary transfer functions.
It is easy to understand, from the above discussion, why neural nets simulated on computers require so much computing power, even on computers specially designed for neural net simulations. It is also clear that the task of physically implementing a large network is quite difficult.
A directly implemented neural network is one in which a physical processor node implements every node in a neural net, that is, a physical processor exists for each node. The problem of interconnection must be dealt with whenever direct implementations of neural nets are built. Any direct implementation of such nets faces four problems which the present invention either overcomes or circumvents.
The first problem is simply that of interconnection. Presently, workers deal with the problem by reducing the interconnectivity of the net so that, for example, each node can only receive data from ten other nodes. This makes it impossible to build many networks including the classic one described above with respect to FIG. 2.
A second problem is that of flexibility. Because neural networks vary so much in structure, and because the structure of a net needed to solve a particular problem is generally unknown at the outset (which is why the nets must "learn"), it is important that any hardware implementation provide variability in net structure.
A third problem is that of construction. Because of the huge size of neural nets, it is also very important that any structure built be very simple and regular. For most present-day implementations, if the net changes slightly, the computing machine must be abandoned (if the nodes are realized on an integrated circuit), or physically and extensively rewired to accommodate the structural changes.
A fourth problem is that of the efficient use of resources. Equations 2 and 3 show a sum of products as part of a node function. In an analog implementation these multiple inputs can be summed simultaneously with the appropriate circuitry. Digital logic is not generally designed to sum many input simultaneously and if attempted would be both expensive and inflexible. Because digital implementations do not process simultaneous inputs, providing simultaneous inputs is inefficient. Hence, a direct implementation of FIG. 1 is inefficient for a digital system.